OWL: A Recommender System for Organization-Wide Learning

We describe the use of a recommender system to enable continuous knowledge acquisition and individualized tutoring of application software across an organization. Installing such systems will result in the capture of evolving expertise and in organization-wide learning (OWL). We present the results of a year-long naturalistic inquiry into application’s usage patterns, based on logging users’ actions. We analyze the data to develop user models, individualized expert models, confidence intervals, and instructional indicators. We show how this information could be used to tutor users. Introduction Recommender Systems typically help people select products, services, and information. A novel application of recommender systems is to help individuals select ’what to learn next’ by recommending knowledge that their peers have found useful. For example, people typically utilize only a small portion of a software application’s functionality (one study shows users applying less than 10% of Microsoft Word’s commands). A recommender system can unobtrusively note which portions of an application’s functionality that the members of an organization find useful, group the organization’s members into sets of similar users, or peers (based on similar demographic factors such as job title, or similarities in command usage patterns), and produce recommendations for learning that are specific to the individual in the context of his/her organization, peers, and current activities. This paper reports research on a recommender system (Resnick & Varian, 1997) intended to promote gradual but perpetual performance improvement in the use of application software. We present our rationale, an analysis of a year’s collected data, and a vision of how users might learn from the system. We have worked with one commercial application, and believe our approach is generally applicable. The research explores the potential of a new sort of user modeling based on summaries of logged user data. This method of user modeling enables the observation of a large number of users over a long period of time, enables concurrent development of student models and individualized expert models, and applies recommender system techniques to on-the-job instruction. Earlier work is reported in Linton (1990), and Linton (1996). Kay and Thomas (1995), Thomas (1996) report on related work with a text editor in an academic environment. A recommender system to enhance the organization-wide learning of application software is a means of promoting organizational learning (Senge, 1990). By pooling and sharing expertise, recommender systems augment and assist the natural social process of people learning from each other. This approach is quite distinct from systems, such as Microsoft’s Office Assistant, which recommend new commands based on their logical equivalence to the lessefficient way a user may be performing a task. The system presented here will (1) capture evolving expertise from community of practice (Lave & Wenger 1991), (2) support less-skilled members of the community in acquiring expertise, and (3) serve as an organizational memory for the expertise it captures. In many workplaces ... mastery is in short supply and what is required is a kind of collaborative bootstrapping of expertise. (Eales & Welch, 1995, p. 100) The main goal of the approach taken in this work is to continuously improve the performance of application users by providing individualized modeling and coaching based on the automated comparison of user models to expert models. The system described here would be applicable in any situation where a number of application users perform similar tasks on networked computers 65 From: AAAI Technical Report WS-98-08. Compilation copyright © 1998, AAAI (www.aaai.org). All rights reserved. In the remainder of this section we describe the logging process and make some initial remarks about modeling and coaching software users. We then present an analysis of the data we have logged and our process of creating individual models of expertise. In the final section we describe further work and close with a summary. Each time a user issues a Word command such as Cut or Paste, the command is written to the log, together with a time stamp, and then executed. The logger, called OWL for Organization-Wide Learning, comes up when the user opens Word; it creates a separate log for each file the user edits, and when the user quits Word, it sends the logs to a server where they are periodically loaded into a database for analysis. A toolbar button labeled ’OWL is ON’ (or OFF) informs users of OWL’s tate and gives them control. Individual models of expertise We have selected the Edit commands for further analysis. A similar analysis could be performed for each type of command. The first of the three tables in Figure 1 presents data on the Edit commands for each of our 16 users. In the table, each column contains data for one user and each row contains data for one command (Edit commands that were not used have been omitted). A cell then, contains the count of the number of times the individual has used the command. The columns have been sorted so that the person using the most commands is on the left and the person using the fewest is on the right. Similarly, the rows have been sorted so that the most frequently used command is in the top row and the least frequently used command is in the bottom row. Consequently the cells with the largest values are in the upper left corner and those with the smallest values are in the lower right comer. The table has been shaded to make the contours of the numbers visible: the largest numbers have the darkest shading and the smallest numbers have no shading, each shade indicates an order of magnitude. Inspection of the first table reveals that users tend to acquire the Edit commands in a specific sequence, i.e., those that know fewer commands know a subset of the commands used by their more-knowledgeable peers. If instead, users acquired commands in an idiosyncratic order, the data would not sort as it does. And if they acquired commands in a manner that strongly reflected their job tasks or their writing tasks, there would be subgroups of users who shared common commands. Also, the more-knowledgeable users do not replace commands learned early on with more powerful commands, but instead keep adding new commands to their repertoire. Finally, the sequence of command acquisition corresponds to the commands’ frequency of use. While this last point is not necessarily a surprise, neither is it a given. There are some peaks and valleys in the data as sorted, and a fairly rough edge where commands transition from being used rarely to being used not at all. These peaks, valleys, and rough edges may represent periods of repetitive tasks or lack of data, respectively, or they may represent overdependence on some command that has a more powerful substitute or ignorance of a command or of a task (a sequence of commands) that uses the command. In other words, some of the peaks, valleys, and rough edges may represent opportunities to learn more effective use of the software. In the second table in Figure 1 the data have been smoothed. The observed value in each cell has been replaced by an expected value, the most likely value for the cell, using a method taken from statistics, based on the row, column and grand totals for the table (Howell, 1982). In the case of software use, the row effect is the overall relative utility of the command (for all users) and the column effect is the usage of related commands by the individual user. The expected value is the usage the command would have if the individual used it in a manner consistent with his/her usage of related commands and consistent with his/her peers’ usage of the command. These expected values are a new kind of expert model, one that is unique to each individual and each moment in time; the expected value in each cell reflects the individual’s use of related commands, and one’s peers’ use of the same command. The reason for differences between observed and expected values, between one’s actual and expert model, might have several explanations such as the individual’s tasks, preferences, experiences, or hardware, but we are most interested when the difference indicates the lack of knowledge or skill.